Concepts: Measuring Quality
Topics
Measuring Quality 
The measurement of Quality, whether Product or Process, requires the collection and analysis of information, usually stated in terms of measurements and metrics. Measurements are made primarily to gain control of a project, and therefore be able to manage it. They are also used to evaluate how close or far we are from the objectives set in the plan in terms of completion, quality, compliance to requirements, etc.
Metrics are used to attain two goals, knowledge and change (or achievement):
Knowledge goals: they are expressed by the use of verbs like evaluate, predict, monitor. You want to better understand your development process. For example, you may want to assess product quality, obtain data to predict testing effort, monitor test coverage, or track requirements changes.
Change or achievement goals: these are expressed by the use of verbs such as increase, reduce, improve, or achieve. You are usually interested in seeing how things change or improve over time, from an iteration to another, from a project to another.
Metrics for both goals are used for measuring Process and Product Quality.
All metrics require criteria to identify and to determine the degree or level at which of acceptable quality is attained. The level of acceptable quality is negotiable and variable, and needs to be determined and agreed upon early in the development lifecycle For example, in the early iterations, a high number of application defects are acceptable, but not architectural ones. In late iterations, only aesthetic defects are acceptable in the application.
The acceptance criteria may be stated in many ways and may include more than one measure. Common acceptance criteria may include the following measures:
- Defect counts and / or trends, such as the number of defects identified, fixed, or that remain open (not fixed).
- Test coverage, such as the percentage of code, or use cases planned or implemented and executed (by a test). Test coverage is usually used in conjunction with the defect criteria identified above).
- Performance, such as a the time required for a specified action (use case, operation, or other event) to occur. This is criteria is commonly used for Performance testing, Failover and recovery testing, or other tests in which time criticality is essential.
- Compliance. This criteria indicates the degree to which an artifact or process activity / step must meet an agreed upon standard or guideline.
- Acceptability or satisfaction. This criteria is usually used with subjective measures, such as usability or aesthetics.
See Concepts: Metrics for additional information.
See Evaluating Quality Throughout the Process for a matrix identifying the phases, workflows, and artifacts in which an artifact or activity is assessed for quality.
Measuring Product Quality
Stating the requirements in a clear, concise, and testable fashion is only part of achieving product quality. It is also necessary to identify the measures and criteria that will be used to identify the desired level of quality and determine if it has been achieved. Measures describe the method used to capture the data used to assess quality, while criteria defines the level or point at which the product has achieved acceptable (or unacceptable) quality.
Measuring the product quality of an executable artifact is achieved using one or more measurement techniques, such as:
- reviews / walkthroughs
- inspection
- execution
Different metrics are used, dependent upon the nature the quality goal of the measure. For example, in reviews, walkthroughs, and inspections, the primary goal is to focus on the function and reliability quality dimensions. Defects, coverage, and compliance are the primary metrics used when these measurement techniques are used. Execution however, may focus on function, reliability, or performance. Therefore defects, coverage, and performance are the primary metrics used. Other measures and metrics will vary based upon the nature of the requirement.
See Concepts: Key Measures of Test for additional information.
See Guidelines: Metrics for additional information.
See Concepts: Product Quality for additional information.
Measuring Process Quality
The measurement of Process Quality is achieved by collecting both knowledge and achievement measures.
- The degree of adherence to the standards, guidelines, and implementation of an accepted process.
- Status / state of current process implementation to planned implementation.
- The quality of the artifacts produced (using product quality measures described above).
Measuring process quality is achieved using one or more measurement techniques, such as:
- progress - such as use cases demonstrated or milestones completed
- variance - differences between planned and actual schedules, budgets, staffing requirements, etc.
- product quality measures and metrics (as described in Measuring Product Quality section above)
See Guidelines: Metrics for additional information.
See Workflow: Introduction to Project Management for additional information.
See Concepts: Process Quality for
additional information.
